Native Jetpack Installation: Apply A/B test #8170

selanthiraiyan · 2022-11-21T09:33:07Z

Closes: #8166

Description

This PR adds a new A/B test experiment for simplified login to enable native Jetpack installation flow. Experiment details:

Added new A/B test experiment.
Replaced the temporary feature flag with A/B test.

Testing instructions

Prerequisite: Make sure that you have a test store that is not associated with your WP.com account or doesn't have Jetpack.

`control` variation

Log out of the app or skip login onboarding if needed.
On the prologue screen, select "Log In" or "Continue with WordPress.com" based on the A/B test variant you get.
Log in with your WordPress.com account.
If your account doesn't have any associated site, notice that the empty store picker is displayed with the matching design as on Figma.
Select Add a Store > Connect an existing site.
Enter the address of your test store and tap Continue.
You will see the existing Jetpack error screen.
You can install and setup Jetpack in a webview from this screen.

Testing the `treatment` variation

We will have to sandbox and assign a particular variation to the device to test the variations.

Prerequisites:

Please follow the setup in PCYsg-Fq7-p2#assignments-api for proxied sandboxed API requests: connect to sandbox via ssh, and point WP.com API to the sandbox IP address (ifconfig) in the local matchine's /etc/hosts
In Xcode, set a breakpoint at ExPlatService L47 where anon_id is set in the URL request
Make an API request PATCH /wpcom/v2/experiments/0.1.0/assignments with the following body and make sure it's successful:

{
    "variations": {
        "woocommerceios_login_jetpack_setup_flow": "treatment"
    },
    "anon_id": "anon_id_from_the_previous_step",
    "username_override": ""
}

Launch the app, and let it sit for a bit
Close and relaunch the app
Log out of the app or skip login onboarding if needed.
On the prologue screen, select "Log In" or "Continue with WordPress.com" based on the A/B test variant you get.
Log in with your WordPress.com account.
If your account doesn't have any associated site, notice that the empty store picker is displayed with the matching design as on Figma.
Select Add a Store > Connect an existing site.
Enter the address of your test store and tap Continue.
You will see the new Jetpack error screen. (Screenshot below)
You can install and setup Jetpack in new native flow starting from this screen.

Screenshots

Control	Treatment

I have considered if this change warrants user-facing release notes and have added them to RELEASE-NOTES.txt if necessary.

wpmobilebot · 2022-11-21T12:09:55Z

You can test the changes from this Pull Request by:

Clicking here or scanning the QR code below to access App Center
Then installing the build number pr8170-f61ea32 on your iPhone

If you need access to App Center, please ask a maintainer to add you.

jaclync

@selanthiraiyan I was testing the treatment case, but I couldn't get the variation after following the setup. The API request to set the assignment manually worked, as I saw the assignments with woocommerceios_login_jetpack_setup_flow: treatment initially for the anon ID. However, when logging in, the app calls ABTest.start(for:) again and that resets the assignments for the logged-in context. Were you able to get the treatment variation from your testing? Maybe we should consider this AB experiment in the logged-in context instead, since merchants can only install Jetpack after logging in?

selanthiraiyan · 2022-11-24T07:26:46Z

@jaclync

Were you able to get the treatment variation from your testing?

I remember getting the treatment variation. I think I tested it incorrectly while raising my PR.

Were you able to get the treatment variation from your testing? Maybe we should consider this AB experiment in the logged-in context instead, since merchants can only install Jetpack after logging in?

I started treating the experiment as logged-in, but so far I am not able to assign the "treatment" variation to my simulator's anonId (device ID).
I tried to debug and noticed that anonId is nil in func getAssignments(completion: @escaping (Assignments?) -> Void) while trying to reload experiment data after logging in. 🤔
I will keep looking into this. Meanwhile, kindly let me know if you can successfully test this by assigning the treatment variation.

jaclync

I tested again with the logged-in paragraph in the assignments API FG linked in the PR description, and it worked for my a8c WPCOM account (somehow it didn't work using a bearer token for a non-a8c account). I verified that the assignments from the API have the treatment variation set for the Jetpack experiment. However, the variant in the app use case stays control because the assignments from the API weren't saved to UserDefaults since the ExPlat instance was deallocated.

After looking into the deallocation of ExPlat a bit, it looks like ExPlat.shared is overridden whenever ExPlat.configure is called. When launching the app in the logged-in state, it's called first with an anon ID and then again in the authenticated state which reset ExPlat.shared so the previous instance was deallocated. The assignments API call was made with the first ExPlat instance, so the assignments value couldn't be saved for access later.

I haven't looked into a solution yet, maybe we'll have to make sure the ABTest assignments API call is made only with the authenticated ExPlat? I also wonder if this might have caused the biased A/A test that we saw earlier pbxNRc-1QS-p2#comment-3819.

jaclync · 2022-11-24T08:56:25Z

Experiments/Experiments/ABTest.swift

+    /// A/B test to measure the sign-in success rate when native Jetpack installation experience is enabled
+    /// Experiment ref: pbxNRc-29W-p2
+    ///
+    case abTestNativeJetpackSetupFlow = "woocommerceios_login_jetpack_setup_flow"


nit: abTest might be redundant in the case name?

WDYT @selanthiraiyan?

Sorry, I missed this. Done in 2121554

itsmeichigo · 2022-11-25T02:25:01Z

Thank you, @jaclync for looking into the issue and sharing the observation! I'm trying to understand the problem here:

I verified that the assignments from the API have the treatment variation set for the Jetpack experiment. However, the variant in the app use case stays control

Does it mean you can get the treatment variation after Sharma sets the Jetpack experiment to logged-in?

After looking into the deallocation of ExPlat a bit, it looks like ExPlat.shared is overridden whenever ExPlat.configure is called. When launching the app in the logged-in state, it's called first with an anon ID and then again in the authenticated state which reset ExPlat.shared so the previous instance was deallocated. The assignments API call was made with the first ExPlat instance, so the assignments value couldn't be saved for access later.

From my understanding, the problem is not about deallocation because the assignments are saved to UserDefaults after getting a response from getAssignments. The issue is that ExPlat was first configured with an anon ID, and then with a token - so the assignment for the same experiment might change. We want to use each experiment in only an unauthenticated or authenticated state, not both.

With that said, if we want to install Jetpack only in the authenticated state, Sharma's fix should fix the issue with the incorrect variant. I'm concerned if we want to support native Jetpack installation for site credential login in the next iteration - we'll need a new experiment for that then.

I also wonder if this might have caused the biased A/A test that we saw earlier

If my above understanding is correct, I think the fix for the crossovers is to reconfigure ExPlat again with anon ID when the user logs out of the app. This ensures the user gets the exact variation they saw before logging in. I encountered once that I got the different variant after logging out of the app, so doing that should fix the problem. WDYT?

jaclync · 2022-11-25T04:06:13Z

Does it mean you can get the treatment variation after Sharma sets the Jetpack experiment to logged-in?

From my understanding, the problem is not about deallocation because the assignments are saved to UserDefaults after getting a response from getAssignments.

I got the treatment variation when inspecting the assignments when the API result comes back here. And then when ExPlat is handling the assignments to save to UserDefaults, it early returned because self was deallocated. Therefore, I always got the control variation from ABTest.abTestNativeJetpackSetupFlow.variation because the assignments weren't saved to UserDefaults.

Just a note that I tested this when launching the app in the logged-in state to repro the issue more easily, instead of launching the app in the logged-out state and then logging in as in the PR testing steps.

If my above understanding is correct, I think the fix for the crossovers is to reconfigure ExPlat again with anon ID when the user logs out of the app. This ensures the user gets the exact variation they saw before logging in. I encountered once that I got the different variant after logging out of the app, so doing that should fix the problem. WDYT?

Other than the crossover issue, in pbxNRc-1QS-p2#comment-3819 (under Bias in performance) Aaron also noticed a bias toward the control variant in the A/A test results while no bias is expected. I wonder if this deallocation issue might be the reason since the control variant is used when no value is found in the UserDefaults assignments. I can look into a fix for the deallocation issue in a bit.

@itsmeichigo were you able to get the treatment variation following the testing steps with the latest commit? If so, please also feel free to go ahead with approval and merge the PR for the release.

itsmeichigo · 2022-11-25T04:29:20Z

were you able to get the treatment variation following the testing steps with the latest commit? If so, please also feel free to go ahead with approval and merge the PR for the release.

I could not test it because I don't have access to sandbox - I'll have to request that soon. So we'll need your approval anyway 😅

…etpack installation happens after logging in.

…library." This reverts commit 56b1f4b.

…is reset.

jaclync

Works as expected now! 🎉 I can get the treatment variant consistently.

Out of the scope of this PR, I wonder if it's worth creating another 2 A/A tests for logged-out and logged-in states for the same release to see if it fixes the bias towards the control variant mentioned in pbxNRc-1QS-p2#comment-3819?

jaclync · 2022-11-25T08:48:07Z

Experiments/Experiments/ABTest.swift

+    /// A/B test to measure the sign-in success rate when native Jetpack installation experience is enabled
+    /// Experiment ref: pbxNRc-29W-p2
+    ///
+    case abTestNativeJetpackSetupFlow = "woocommerceios_login_jetpack_setup_flow"


WDYT @selanthiraiyan?

selanthiraiyan added type: enhancement A request for an enhancement. feature: login Related to any part of the log in or sign in flow, or authentication. labels Nov 21, 2022

selanthiraiyan added this to the 11.4 milestone Nov 21, 2022

selanthiraiyan requested review from itsmeichigo and jaclync November 21, 2022 11:18

selanthiraiyan mentioned this pull request Nov 21, 2022

Login: Native experience for the Jetpack installation flow #8075

Closed

15 tasks

selanthiraiyan marked this pull request as ready for review November 24, 2022 03:40

jaclync self-assigned this Nov 24, 2022

jaclync reviewed Nov 24, 2022

View reviewed changes

selanthiraiyan added 5 commits November 25, 2022 10:48

Add Jetpack setup flow AB test case to ABTest enum

526365f

Replace nativeJetpackSetupFlow feature flag with AB testing enum case.

6a6d6e9

Load treatment only when the abTest is not equal to control.

ccb2476

Add experiment reference P2.

305441e

Treat abTestNativeJetpackSetupFlow as logged in experiment as the J…

198414f

…etpack installation happens after logging in.

selanthiraiyan force-pushed the feat/8166-native-jetpack-installation-ab-testing branch from 3c903d5 to 198414f Compare November 25, 2022 05:19

selanthiraiyan and others added 3 commits November 25, 2022 11:43

Update ABTest for the new customTreatment case from Tracks library.

56b1f4b

Revert "Update ABTest for the new customTreatment case from Tracks …

6600bf7

…library." This reverts commit 56b1f4b.

Refresh A/B tests after each refreshUserData where ExPlat.shared …

a90b9e5

…is reset.

jaclync approved these changes Nov 25, 2022

View reviewed changes

selanthiraiyan added 2 commits November 25, 2022 14:32

Remove redundant abTest name from enum case.

2121554

Remove unused feature flag.

f61ea32

selanthiraiyan enabled auto-merge November 25, 2022 09:21

selanthiraiyan merged commit 848f8a7 into trunk Nov 25, 2022

selanthiraiyan deleted the feat/8166-native-jetpack-installation-ab-testing branch November 25, 2022 09:29

jaclync mentioned this pull request Nov 29, 2022

Update logged-out A/A test name and re-add logged-in A/A test #8250

Merged

1 task

selanthiraiyan mentioned this pull request Jan 27, 2023

[REST API] Introduce A/B test and remove feature flag #8744

Merged

1 task

Native Jetpack Installation: Apply A/B test #8170

Native Jetpack Installation: Apply A/B test #8170

Uh oh!

Conversation

selanthiraiyan commented Nov 21, 2022 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Description

Testing instructions

control variation

Testing the treatment variation

Screenshots

Uh oh!

wpmobilebot commented Nov 21, 2022 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

jaclync left a comment

Choose a reason for hiding this comment

Uh oh!

selanthiraiyan commented Nov 24, 2022

Uh oh!

jaclync left a comment

Choose a reason for hiding this comment

Uh oh!

jaclync Nov 24, 2022

Choose a reason for hiding this comment

Uh oh!

jaclync Nov 25, 2022

Choose a reason for hiding this comment

Uh oh!

selanthiraiyan Nov 25, 2022

Choose a reason for hiding this comment

Uh oh!

itsmeichigo commented Nov 25, 2022

Uh oh!

jaclync commented Nov 25, 2022

Uh oh!

itsmeichigo commented Nov 25, 2022

Uh oh!

jaclync left a comment

Choose a reason for hiding this comment

Uh oh!

jaclync Nov 25, 2022

Choose a reason for hiding this comment

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

5 participants

selanthiraiyan commented Nov 21, 2022 •

edited

Loading

`control` variation

Testing the `treatment` variation

wpmobilebot commented Nov 21, 2022 •

edited

Loading